1. Field of the Invention
This invention is related to electronic circuitry on a network, and more particularly, to performing direct memory access (DMA) over the network from one node to another.
2. Description of the Related Art
Networking of computers and other electronic devices has become ubiquitous. While a variety of networking standards exist, Ethernet is one of the most popular. In particular, Gigabit Ethernet and 10 Gigabit Ethernet are becoming widely used. Various protocols are built on top of the network standard, using the network to communicate. The most popular suite of protocols is typically referred to as the transmission control protocol/Internet protocol (TCP/IP).
Communications between devices on the network generally involves providing data to be transmitted, usually prepended with a header that is application-specific. As the data travels down the protocol stack (from TCP to IP to Ethernet to physical transmission) additional header and/or trailer information is added by each layer until the packet transmitted on the network media is formed. The process of adding header/trailer information to a packet is referred to as encapsulation. When a packet is received at the destination, the packet travels up the protocol stack (from Ethernet to IP to TCP to application). Each layer strips off its header and/or trailer data, processes the data, and forwards the packet up to the next layer (assuming no errors are found).
While the protocol stack provides for communication between networked devices, it is not very efficient for moving large amounts of data. For example, processing the packet often involves copying the packet into memory in a receiving device, and then copying the received data to the application's memory space. This extra movement of data is not much of an issue if the amount of data is small, but can be a significant performance issue if the amount of data is large.
One attempt to make data movement more efficient over a network is the remote DMA (RDMA) protocol proposed by the RDMA Consortium. Generally, DMA refers to the direct access by circuitry to memory independent of a central processing unit (CPU) during the access. The CPU may generally be involved in programming the DMA circuitry, but the access itself is performed by the DMA circuitry. RDMA provides a protocol for one network device to directly access memory in another network device without CPU involvement in the actual access.
The RDMA protocol is specified above the TCP layer in the TCP/IP protocol. Thus, RDMA operations travel from the top of the protocol stack to the bottom on the transmitter side, and then up the protocol stack to the top on the receiver side. Typically, RDMA circuitry includes building rudimentary TCP/IP processing hardware that can receive a packet, terminate TCP/IP, process the packet through TCP/IP to the application layer, and extract the data and address at the application layer for writing to memory. The RDMA protocol can prevent the copying of the packet to memory (and then subsequently copying the data payload to memory), making the data transfer more efficient (especially for larger data payloads). However, implementing the RDMA protocol is expensive in terms of hardware overhead (e.g. building TCP/IP processing into hardware) as well as software overhead.